The availability of powerful computer processors at relatively low prices has resulted in many recent methods and systems for processing and manipulating images such as photographs. Computer program-based editing tools are available, for example, that allow two-dimensional images including photographs to be manipulated or edited. Images may be cropped, rotated, skewed in one or more directions, colored or un-colored, and the brightness changed, to name some of the example manipulations that can be made. Images may also be “cut and pasted,” wherein a selected portion of one image is superimposed over a selected portion of a second image. Another known method is so-called “in-painting,” in which an image is extended across regions that have been left blank after removing an unwanted object. Image in-painting typically draws the samples to be filled into blank regions of an image from another portion of the image, and solves a system of partial differential equations to naturally merge the result.
It is also known to analyze a two-dimensional representation of a three dimensional surface to obtain attributes of the three-dimensional surface. For example, so called “shape from shading” methods are known for reconstructing a three-dimensional surface based on the shading found in a two dimensional representation of the original surface. Generally, shape from shading methods recreate a surface by assuming that bright regions of the two-dimensional representation face toward a light source and darker regions face perpendicular or “away” from the light source. Thus a per-region surface normal can be estimated. Reconstruction of a surface from these recovered per-region surface normals, however, can lead to inconsistencies. Shape from shading methods are therefore most often presented in an optimization framework wherein differential equations are solved to recover the surface whose normals most closely match those estimated from the image.
So-called “texture synthesis” is also known, wherein a two-dimensional texture sample is use to generate multiple new, non-repetitive texture samples that can be patched together. By way of example, a photograph of a small portion of a grass lawn can be used to generate a much larger image of the lawn through texture synthesis. Instead of simply repeating the small sample image, texture synthesis can employ a machine learning or similar technique to “grow” a texture matching the characteristics of the original. Each newly “grown” pixel in the synthesized texture compares its neighborhood of previously “grown” pixels in the synthesized texture with regions in the original texture. When a matching neighborhood is found, the newly grown pixel's color is taken from the corresponding pixel in the matching neighborhood in the original texture. Examples of texture synthesis methods include “Pyramid-Based texture analysis/synthesis,” by Heeger et al., Proceedings of SIGGRAPH 95 (1995) 229-238; “Multiresolution sampling procedure for analysis and synthesis of texture images,” by DeBonnet, Proceedings of SIGGRAPH 97 (1997) 361-368; and “Synthesizing natural textures,” by Ashikhmin, 2001 ACM Symposium of Interactive 3D Graphics (2001), all of which are incorporated herein by reference.
Recent texture synthesis work includes “Image Quilting for Texture Synthesis and Transfer,” by Alexei A. Efros and Willian T. Freeman Proc. SIGGRAPH (2001) and “Graphcut textures: Image and video synthesis using graph cuts”, by Kwatra, V. et al. Proc. SIGGRAPH (2003) (“the Graphcut reference”), also incorporated herein by reference. These methods find seams along which to cut to merge neighboring texture swatches so the transition from one swatch to another appears realistic (e.g., the seam falls along the boundary of texture features).
Texture synthesis can be applied to surfaces if there is already a 3-dimensional representation of the surface, for example see “Texture Synthesis on Surfaces” by Greg Turk, Proc. SIGGRAPH (2001), and “Texture Synthesis over Arbitrary Manifold Surfaces” by Li Yi Wei and Marc Levoy, Proc. SIGGRAPH (2001). Also, it is known to apply shape-from-shading to construct a 3-dimensional geometric representation approximating a two-dimensional photographed surface, and then performing texture synthesis on that 3-dimensional geometric representation. However, this method leads to visual artifacts due to the error of reconstructing a globally consistent surface from locally estimated surface normals.